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ABSTRACT 

Two implicit mandates are posited as fundairental to 
the consideration of any successful evaluation procedure by every 
instructor in the basic speech course* First, a clear philosophical 
position must be taken for guiding both the course objectives and the 
evaluation techniques. Second, a successful method for executing the 
philosophical position must be implemented. Described in the article 
are three forms of tests currently being developed at Wichita State 
University for use in the basic course. Written tests which meet 
certain criteria in development and which are matched to a philosophy 
conducive to developing and evaluating communicative competencies 
become the most valuable part of the basic course. (EE) 
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Every acn.iemio rii r,r>i pi i (^xpresfi^^s concern over evaluation proce'Uires 
used in the classroom. Communication instructors seem to feel they have 
special circumstances that require more concern than others. Only a 
casual review of the literature is enough to find large quantities of 
material across a broad spectrum of problems, solutions, and alternatives 
for evaluation procedures in the basic oral communication classroom. 
Discussion has centered on oral communication evaluations, ^ evaluations of 
oral communication, grading by audience reactions,^ grading by audience,^ 
grading by the numbers, ^ grading by written tests, ^ grading by improvement, 
grading by contract,^ grading, ^ and no grading^° — just to cite a few. 

In all of this literature, two implicit mandates can be found as funda- 
mental to the consideration of any successful evaluation procedure: (l) 
a clear philosophical position must be taken for guiding both the course 
objectives and the evaluation techniques, and (2) a successful method for 
executing the philosophical position must be implemented! Eventually these 
two mandates must be considered by every instructor in the basic course. 

Recently, Wichita State University was asked to reevaluate their entire 
program for oral communication which, among other things, allowed for 
reconsideration of a philosophy of evaluation and methods for executing a 
philosophy in two types of basic oral communication courses- the public 
speaking course and the interpersonal communication course. After much 
consideration of the theoretic as well as pragmatic consequences of 
alternative evaluation procedures, we chose to include extensive use of 



written tests in the oral communication program. Beginning in the P'all, 

1973 f term we will employ written tests for exemption and exiting from the 

course and for providing additional instruction within the course. The 

purpose of this presentation is to explain why we arrived at our decision 

and how we intend to execute the use of written tests in the basic course. 

Two preliminary statements should be made about evaluation by testing. 

First, any evaluation procedure is an inference. The closer the inference 

maps reality the better the evaluation procedure. For evaluation of 

learners, the common inferential methods call on judgments that are 

12 

inherently limited. To compensate for these limitations, an accumulation 
of evidence from a variety of testing procedures is the best base for 
framing final evaluations. Therefore, we do not do away r^ith other forms 
of assessment, in particular those of oral performances in the basic course, 
nor do we advocate reliance for assessment solely on results of written 
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testing devices. Second, the best judgments are made from objective data. ^ 
To the extent that all evaluation procedures call for, and rely on, the 
production of objective data , the tetter the inference process and the 
better the base for framing final evaluations. Therefore, we attempted to 
find those procedures that tend to produce objective data. 

II 

Tnere are any number of acceptable philosophical constructs for the 
design of the basic course that lend arguments conducive to defending the 
use of written tests. Three such constructs are given here: the skill 
versus art, performance versus competence, and accountability issues. 

A considerable number of speech professionals prefer to consider 
speaking as an art form. Although somewhat loosely defined, at least in 



textbook discussions, the speaker as artist "successfully matches idea, 

medium, and purpose, utilizing all relevant resources to endow ideas with 
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just the qualities his puirpose requires." Art is distinguishable from 
skill by distinctly "implying a personal, unanalysable creative power. ""'"^ 

Although the preference for speech as art is admirable (and a 
preference difficult to challenge at face value), it is an inappropriate 
choice for those courses required to be taken by all university students. 
Course objectives based on measures of artistic merit as framed by the 
previous definitions cannot be achieveri by even a small number of students. 
In particular, the basic course is not appropriate ground for the develop- 
ment of polished orators. However, the most significant argument is that 
evaluation by artistic elements does not yield to standardized analyzable 
data. If the basic course is offered in multiple sections, in recurring 
semesters, by differing sets of instiructors, as a requirement for all 
students in the university, the failure to identify and standardize criteria 
for evaluation will reduce the basic-course program to chaos. 

This does not argue that the other side of the coin, the skills 

approach, is to be chosen by default. Such an approach can stand on its 

own merit by adopting a more sophisticated analysis for framing the design 

of a basic course. This analysis can be made by drawing metaphorically on 

the linguists* performance-competence dichotomy. For linguistics, the 

distinction of performance and competence is crucial. Competence is the 

set of rules for language behavior while performance is the language 

behavior. As DeVito notes: "performance is determined by many factors 
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only one of which is competence," A direct application of performance- 
competence to communication has been attempted by some very recent authors. 



For example, Robert Nof singer feels "there is reason to believe that there 
may be a rule structure to communication, and some scholars believe that 
it is possible to gather evidence which should firmly establish or refute 
this assertion. "■'■^ 

If the dichotomy is transferred as a metaphor, then a clear distinction 
exists between discovering and internalizing rules for effective communi- 
cation behavior and engaging in patterns of communication behavior. The 
recognition and capacity for effective public speaking, or interpersonal 
communication, is communication competence while speaking or interacting 
is performance. The question of whether rules for determining communication 
competence exist to the extent rules ejfist for linguistic competence is 
certainly beyond the scope of this paper and is the major reason for seeking 
a metaphorical mapping. However, to the extent an instructor accepts a 
rhetorical uheory, establishes rules for effective speaking, or advocates 

determinates of effective interpersonal behavior, that instructor is 
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attempting to locate a communication competency, To the extent the 
instructor expects the student to know and be able to apply the theory, 
rules, or determinates, the instructor is expecting a reflection of 
communication competence. 

The linguist plau s a great ajnount of importance upon the appraisal 
of linguistic competence. The speech teacher should place an equal a'aount 
of importance on the appraisal of communication competence. The process 
of evaluating communication competence must be carefully handled to assure 
that distinction is made between competence and performance. 

It is dubious, if not wholly impossible, to determine an accurate 
measure of communication competence by testing oral performances. First, 



the oral act is an exceedingly complex set of behaviors interacting in an 
almo t infinite number of patterns, iflien the evaluator enters the communi- 
cation system the complexity and variety increases even further. Research 

has rather conclusively established the lack of validity in assessing oral 
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performances even when attempts to standardize evaluation have been used. 
What past researchers have told us is that factors of attention, selectivity, 
fatigue, emotional state, prior speeches given by the class, prior speeches 
given by the particular student, expectancies, personal likes and dislikes, 
and countless others interact to make oral evaluations a seriously limited 
evaluation procedure. Second, the testing for competence in the oral 
performance works to the advantage of the student who has talented verbal 
facility. Although research may prove me wrong, verbal facility seems the 
single most dominant characteristic to mask inadequacies of oral presen- 
tations. In the typical speech class, whore evaluations are ba.,ed on 
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comparisons, the verbally talented will be advantaged. Third, the 
classroom experience is supposed to represent a simulated experience 
generalizable to typical communication experiences of the student's current 
and future environment. As long as any evidence exists that the classroom 
oral communication experience demands unique behavior patterns not generaliz- 
able to typical (non- simulated) experiences then evaluation of the 
experience is meaningless. As we are only beginning to docui^ient in ^^rban 
settings with inner-city or ghetto students, the classroom experience 
tests performance that is not reflective of competence. Conville and Story 
have recently claimed that classroom communication behavior better reflects 

a student's ability "to defend [_himGelf] in a hostile and thit^^tening 

22 

situation" than to competently communicate. 
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A serious issue facing all facets of academic life is "accountability." 
Although typically defined in terms of "cost-accounting," the concept 
has ramifications beyond simple economic questions. Students and admini- 
strators are almost equal with legislators in demands for an accounting of 
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educational procedures and evaluation techniques. This issue must be 

taken seriously for the power really rests with those asking the questionSf 

and the typical basic course is in no position to evidence answers. Until 

the basic course moves to established course criteria in which objective data 

is used to confirm learning achievement, there can be challenges (and " 

losses) in the speech curriculum. 

One approach to the demands of accountability is the use of behavioral 

objectives. Behavioral objectives allow for consideration of the 

appropriate concepts and content for the course as well as a description of 

"what students will be able to do after completing a prescribed unit of 

instruction."^^ Descriptions and recommendations for implementing 
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behavioral objectives are araply documented in the literature. Here, we 
would note behavioral objectives* major advantage over other schemes is 
they allow components of complex behavior to be broken not only into con- 
stituent elements of behavior but into levels of behavior. For example, 
a requirement of "a five minute speech to inform" involves a rlPinoniilrctKiou 
of behavior in the psychomoter, affective, and cognitive behavior domains 
as well as a demonstration of behavior at different levels of each domain. 
If the behavioral objectives do discern more specific components and levels 
of behavior and if those discernments call on cognitive handling of 
communication competencies, then written tests can and should be an 
appropriate ni^>?\nurin^ hor'.hinmie for evaluating achievement of the objectives. 
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The previous arguments of this paper suggest that the oral conununi- 
cation course, public speaking or ? ''teirpsrsonal communication, for the . 
modern university or college has be -Ognize priorities for teaching 
communication competencies in an accountable manner. Therefore, it follows 
that such a course not only can make use of written testing instruments 
but should make use of written testing instruments, 

III 

The fundamental distinction between testing devices of any kind centers 

on the procedure by which achievement is measured. There are two common 

procedures: norm-referenced measurement (NRfl) and criterion-referenced 

measurement (CRM). Norm-referenced measurement "is designed to ascertain a 

student's performance in relation to the performance of others on the same 
27 

test," Criterion- referenced measurement is "designed to determine whether 

a student has achieved mastery of a behavior as specified in a performance 
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objective at a given criterion level," Recently, an excellent article by 
Smythe, Kibler, and Hutchings ^ detailed the difference between norm- and 
criterion-referenced measurement and suggc-.ted implications of criterion- 
re xorenced measurement for communication instruction. This is an important 
article for those instructors whose courses follow the communication 
competencies model since criterion-referenced measurement is the most 
appropriate measuring procedure. Instructors using behavioral objectives 
will also be interested in criterion-referenced measurement as the only 
useful measuring procedure. 

For the WSU program, all uses of written tests will be based on the 
criterion- referenced measurement procedure. Issues that become critical 
in adopting such a procedure are the same as for any evaluating procedure: 
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validity, reliability, usability, and power. Validity, tho ability of the 
test to measure what it intends to measure, and reliability, the ability 
of the test to measure consistently, are issues that have already been 
discussed, and resolved in favor of written testing over evaluation of oral 
performances for the communication compctv^ncies model. Usability is the 
aoility of the test to effecicntly meet the d^ands and objectives of the 
course. If the test is practical in t^rms of time limits and pconomy, is 
clearly understood by students in terms of the relevant cbjcc hives, and is 
appropriate for the type of learning that is being measured the test is 
usable. Power is the ability of the test to challenge and discriminate 
among those who have competence and those who do not. All four issues are 
met as much in the design of behavioral objectives, in particular the setting 
of criteria for the objectives, as they are in the design of the test. 
Since this paper is more descriptive of tests than objectives, we 
ask for consideration of three possible uses for written tests that follow 
the philosophy of a course built around communication competence as 
reflected in behavioral objectives. VJSU is currently in the process of 
developing and implementing all three uses for written tests. 

WSU is under the kind of pressure felt by others to have a procedure 
by which students can be exempted from taking a required course. The con- 
cept of an exemption device is based on the assumption that some students 
already possess the abilities expected upon completion of the course; 
therefore, they have little to gain from taking that course. While dis- 
tasteful for some o believe they have nothing to offer some students, the 
argument is presentable if one has identified specific objectives for 
his course and expects that behavior demonstrating mastery of those objectives 



9 



is sufficient for satisfying the demands of the courc^, Tf tho shnrl.mh <vnn 

meet the demands of the course, why should he have tu take the courec? 

The real problem, however, is discovuring the procedure best able to 

predict the student can meet those demands. A number of researchers have 

f3iT^cuica bhoi-r -x-t-fcou+T-n *-o dirovvorinG ^ pror;orluro for predicting success 

in the basic course, Judd and Smith^° and Wall^-"" have tried to corrolate 

SAT Verbal Scores (as well as other measures) with success in the basic 

course, but neither study would firmly stats that such scores wor^^ sufri^-jniL 

as predictors. Kibler, Kelly, Gibson, and Gmnci:^^' reported a significant 

correlation between the number of syllables used by students in a public 

speech sample and the final grade. However, from a political position 

within the univ( .^sity, we do not feel it would be prudent to inform the 

University Advisory Board that students will be exempt from the basic course 

based on a sajnpling of their speech that shows a high syllable count. 

This research does not rule out the findings of an effective exemption 

procedure. Ellis provides some evidence in his reporting of a relatively 

successful vjritten test consisting of throe subtests: (l) a measure of the 

student's prior speaking experience, (2) a self- rating of speech skills 
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and fears, and (3) the student's liking for speech- related activities. 
This gave us confidence in the potential for paper-and-pencil testing but 
gave us caution since we were looking for some measure of cognitive 
abilities rather than affective behavior. For a measurement of cognitive 
behavior, we turned to an examination used by Pennsylvania State University 
for predicting who should be exempted from their basic program. Their 
noted success with their test, especially using paper-and-pencil screening 
only, was sufficient to argue for a similarly constructed test at WSU. 
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The second use for written tests developed as the most fsasible 
answer to a scries of problems. The basic course had been under severe 
criticise to the point the course was in danger of being dropped from the 
cuzrriculum. V?ith forty-five sections offered each semester there was not 
only a considerablti difference in what the students believed were the 
course objectives, there was a considerable difference in what they repox-ted 
were the ooursj requirements. The content and requlx-emenh vaT-janco botx^v;^ii 
instructors teaching the course was so extensive, there wa.s no single basic 
speech course. The second problem was the course had no device to document, 
for accountability purposes, the student's satisfaction of the course 
objectives. Among the methods for resolving those problems was to devise 
a uniform exiting device. This means that at the end of each semester, all 
the students enrolled in all sections of the basic course would take a uni- 
form final examination. 

The construction of such a device will be consistent with the goals, 

philosophies, and requirements for a test for this kind of a course. The 

device will be constructed by using a matrix for determining the number of 
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items appropriate for each cognitive behavioral objective to be tested. 
Figure I shows such a matrix. Each item in the columns refer to the 
behavioral objective to be tested and each item in the rows refers to the 
level that objective is to be tested at. At the intersection of the rows 



INSERT FIGURE I ABOUT HERE 



and columns is the percentctge of items in the test that will be devoted to 
each objective at each level. This method allows us to construct a 
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balanced test. Furthermore, after the test an evaluation of the instruc- 
tion x"or each instructor for each objective at each level as well as the 
achievement for each student for each objective at each level can be 
determined. 

Written tests can also take the form of sclf-ins' . devices. 

As the third means of using written tests, 'iSU will develop a special 
program of self-instructional material. The' puirpose of such material is to 
aid the student having difficulty in grasping some of the fundamental con- 
cepts. Any number of good program- learning texts are available for the 
basic oral communication course to meet the needs of this student. In 
addition, we will develop special self-instructional programs using computer 
terminals. 

Interactional computer facilities that allow "hands-on" operation by 
students are valuable self-instructional tools. Typically, the operation 

of the facility is simple, thereby requiring little training of the student 
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in the operation procedures. These programs are especially useful for 

drill-and- practice, tutorial work, or simulated exercises. Sophisticated 

programs can divert the student to subroutines that tailor the level of the 
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material to the knowledge level shown by the student's answers, Vfith 
the teletype terminals a permanent record and a copy is available for review 
by the student after leaving the machine and for evaluation by the instructor. 
Such instructional testing devices bring meaning to Benjamin Bloom's defini- 
tion of testing as "the act of gathering and processing evidence about human 
behavior under given conditions for purposes of understanding, predicting, 
and controlling future human behavior," 
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IV 

Written tests can be an effective part of the basic, coiarcfi. If the 
tests meet cortai' -x-" .:ia in development and are matched, to a philoRophy 
conducive to developing and evaluating communication conpetoncies, then 
written tests are a most valuable part of the basic course. The purpose of 
this paper was to demonstrate some of the essential criteria for developing 
and iraplemonting written tosKs, nnd to dor-orlh^^ thr«o f.>Tmr. ..l' wj U+o,, 
tests being developed at 'Jichita State University. 
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SAMPLE MTRIX FOR DETERMINING THE PERCENTAGE COMPOSITION 
OF AN OBJECTIVE TYPE COURSE EXAMINATION FOR EACH COGNITIVE 
BEHAVIORAL OBJECTIVE AT EACH LEVEL OF THE COGNITIVE EOKiAIN 
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